Finiteness Results for Sigmoidal "Neural" Networks

1 1+x 2. This shows that arbitrary (not exp-ra denable) analytic functions may result in architectures with in-nite VC dimension. (Moreover, the architecture used is the simplest one that appears in neural nets practice.) Note that if we wish the x i 's to be bounded, for instance to be restricted to the interval [01; 1], one may replace the above x i 's and w j 's by xi c and cw j , where c = P jx i j. Similarly, if one wants to restrict the weights w j to be bounded, one can use cx i and wj c , with c = P jw j j. Thus bounded weights or inputs (but not simultaneously), even with analytic activations, do not suce. Finally, consider a function f as above, and let : IR ! IR be 0 at 0 and equal to e 01=x 2 elsewhere. Write h(x) := (x)f(1 x): Note that h is even, bounded, has bounded derivative, and is smooth. Thus h can be used in the above constructions instead of f; consider the architecture that results. s are so that the original matrix f(w j x i) had all columns of distinct signs, the same is true of (^ w j ; ^ x i) with the new \f", where ^ w j = 1=w j and ^ x i = 1=x i. The inputs and weights are now all in the interval (0; 1). Starting with f = cos, this illus-tates that even with bounded weights and inputs, in-nite dierentiability is not sucient to guarantee nite VC dimension. generalizations of the PAC model for neural net and other learning applications ," Inform. Proving the existence of universal teaching sets of cardinality 2r + 1, and in fact that almost all sets of that cardinality are universal teaching sets, is now easy. Indeed, consider a new architecture with weight space IR 2r and such that the new behavior satises 0 ((w 1 ; w 2); x) := (w 1 ; x) 0 (w 2 ; x): Fix any w 0. Then, any teaching set for (w 0 ; w 0), that is, a teaching set for the identically zero function, is a universal teaching set. Since the parameter space is now of dimension 2r, the result follows. Remark 5.4 It can be shown by examples that the bounds are best possible. Also, …